<html>
<head><meta charset="utf-8"><title>consider storing commit CI builds longer · t-infra · Zulip Chat Archive</title></head>
<h2>Stream: <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/index.html">t-infra</a></h2>
<h3>Topic: <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/consider.20storing.20commit.20CI.20builds.20longer.html">consider storing commit CI builds longer</a></h3>

<hr>

<base href="https://rust-lang.zulipchat.com">

<head><link href="https://rust-lang.github.io/zulip_archive/style.css" rel="stylesheet"></head>

<a name="215256462"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/consider%20storing%20commit%20CI%20builds%20longer/near/215256462" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Noah Lev <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/consider.20storing.20commit.20CI.20builds.20longer.html#215256462">(Nov 01 2020 at 18:45)</a>:</h4>
<p>Recently when bisecting I've had several cases where the regressed commit was more than 167 days ago, and thus I can't bisect to a specific commit. Could we store CI builds longer, e.g. for a year instead of 167 days?</p>



<a name="215256538"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/consider%20storing%20commit%20CI%20builds%20longer/near/215256538" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Joshua Nelson <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/consider.20storing.20commit.20CI.20builds.20longer.html#215256538">(Nov 01 2020 at 18:46)</a>:</h4>
<p>storing CI builds is expensive</p>



<a name="215256549"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/consider%20storing%20commit%20CI%20builds%20longer/near/215256549" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Joshua Nelson <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/consider.20storing.20commit.20CI.20builds.20longer.html#215256549">(Nov 01 2020 at 18:47)</a>:</h4>
<p>how much benefit would you expect to get compared to 'between these two nightlies'?</p>



<a name="215256721"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/consider%20storing%20commit%20CI%20builds%20longer/near/215256721" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Noah Lev <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/consider.20storing.20commit.20CI.20builds.20longer.html#215256721">(Nov 01 2020 at 18:51)</a>:</h4>
<p>Well it's helpful to find the exact commit that caused the problem</p>



<a name="215256961"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/consider%20storing%20commit%20CI%20builds%20longer/near/215256961" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> simulacrum <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/consider.20storing.20commit.20CI.20builds.20longer.html#215256961">(Nov 01 2020 at 18:57)</a>:</h4>
<p>Can you give some examples where this is useful? Note that we  store nightlies indefinitely</p>



<a name="215256979"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/consider%20storing%20commit%20CI%20builds%20longer/near/215256979" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> simulacrum <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/consider.20storing.20commit.20CI.20builds.20longer.html#215256979">(Nov 01 2020 at 18:57)</a>:</h4>
<p>and it should not be too hard to rebuild the 5-6 commits from back then if really necessary and can't otherwise be identified</p>



<a name="215258009"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/consider%20storing%20commit%20CI%20builds%20longer/near/215258009" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Noah Lev <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/consider.20storing.20commit.20CI.20builds.20longer.html#215258009">(Nov 01 2020 at 19:23)</a>:</h4>
<p>I had this issue when bisecting <a href="https://github.com/rust-lang/rust/issues/78632">#78632</a> and some other issues as well</p>



<a name="215258058"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/consider%20storing%20commit%20CI%20builds%20longer/near/215258058" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Noah Lev <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/consider.20storing.20commit.20CI.20builds.20longer.html#215258058">(Nov 01 2020 at 19:24)</a>:</h4>
<p>Sometimes the bug isn't reported until several months after it's introduced, in which case I can't bisect to a specific commit.</p>



<a name="215258069"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/consider%20storing%20commit%20CI%20builds%20longer/near/215258069" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Noah Lev <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/consider.20storing.20commit.20CI.20builds.20longer.html#215258069">(Nov 01 2020 at 19:24)</a>:</h4>
<p>I'm just wondering if we could store the commit builds longer</p>



<a name="215258880"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/consider%20storing%20commit%20CI%20builds%20longer/near/215258880" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Joshua Nelson <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/consider.20storing.20commit.20CI.20builds.20longer.html#215258880">(Nov 01 2020 at 19:45)</a>:</h4>
<p>I think I found the relevant PR</p>



<a name="215258894"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/consider%20storing%20commit%20CI%20builds%20longer/near/215258894" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Joshua Nelson <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/consider.20storing.20commit.20CI.20builds.20longer.html#215258894">(Nov 01 2020 at 19:45)</a>:</h4>
<p>clearly there's <em>some</em> benefit to storing the builds longer, but I don't think it's clear the benefit is worth the storage cost</p>



<a name="215259466"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/consider%20storing%20commit%20CI%20builds%20longer/near/215259466" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> simulacrum <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/consider.20storing.20commit.20CI.20builds.20longer.html#215259466">(Nov 01 2020 at 19:59)</a>:</h4>
<p>I think 167 days is plenty for most cases. I do not think that I would want to raise the limit even more right now</p>



<a name="215261144"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/consider%20storing%20commit%20CI%20builds%20longer/near/215261144" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Pietro Albini <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/consider.20storing.20commit.20CI.20builds.20longer.html#215261144">(Nov 01 2020 at 20:35)</a>:</h4>
<p><span class="user-mention" data-user-id="307537">@Camelid</span> a thing to consider is, last time I checked every commit is around 27GB of artifacts</p>



<a name="215261148"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/consider%20storing%20commit%20CI%20builds%20longer/near/215261148" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Noah Lev <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/consider.20storing.20commit.20CI.20builds.20longer.html#215261148">(Nov 01 2020 at 20:35)</a>:</h4>
<p>Wow, that's a lot</p>



<a name="215261193"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/consider%20storing%20commit%20CI%20builds%20longer/near/215261193" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Noah Lev <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/consider.20storing.20commit.20CI.20builds.20longer.html#215261193">(Nov 01 2020 at 20:36)</a>:</h4>
<p>Maybe we could just keep rustc, std, and cargo? Though I guess you have to store a copy for each target. Might not be worth it</p>



<a name="215261194"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/consider%20storing%20commit%20CI%20builds%20longer/near/215261194" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Pietro Albini <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/consider.20storing.20commit.20CI.20builds.20longer.html#215261194">(Nov 01 2020 at 20:36)</a>:</h4>
<p>and storing those 167 days of commits is already the biggest item in our bill <span aria-label="sweat smile" class="emoji emoji-1f605" role="img" title="sweat smile">:sweat_smile:</span></p>



<a name="215261198"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/consider%20storing%20commit%20CI%20builds%20longer/near/215261198" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Pietro Albini <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/consider.20storing.20commit.20CI.20builds.20longer.html#215261198">(Nov 01 2020 at 20:36)</a>:</h4>
<p>yeah it's for all targets</p>



<a name="215261206"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/consider%20storing%20commit%20CI%20builds%20longer/near/215261206" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Noah Lev <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/consider.20storing.20commit.20CI.20builds.20longer.html#215261206">(Nov 01 2020 at 20:37)</a>:</h4>
<p>Who pays for infra btw?</p>



<a name="215261211"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/consider%20storing%20commit%20CI%20builds%20longer/near/215261211" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Joshua Nelson <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/consider.20storing.20commit.20CI.20builds.20longer.html#215261211">(Nov 01 2020 at 20:37)</a>:</h4>
<p><span class="user-mention silent" data-user-id="121055">Pietro Albini</span> <a href="#narrow/stream/242791-t-infra/topic/consider.20storing.20commit.20CI.20builds.20longer/near/215261194">said</a>:</p>
<blockquote>
<p>and storing those 167 days of commits is already the biggest item in our bill <span aria-label="sweat smile" class="emoji emoji-1f605" role="img" title="sweat smile">:sweat_smile:</span></p>
</blockquote>
<p>since this is mostly only for bisecting, would it make sense to only store the top 3 targets? windows/linux/mac?</p>



<a name="215261212"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/consider%20storing%20commit%20CI%20builds%20longer/near/215261212" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Noah Lev <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/consider.20storing.20commit.20CI.20builds.20longer.html#215261212">(Nov 01 2020 at 20:37)</a>:</h4>
<p>Is it different companies who sponsor Rust?</p>



<a name="215261213"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/consider%20storing%20commit%20CI%20builds%20longer/near/215261213" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Pietro Albini <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/consider.20storing.20commit.20CI.20builds.20longer.html#215261213">(Nov 01 2020 at 20:37)</a>:</h4>
<p>aws</p>



<a name="215261215"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/consider%20storing%20commit%20CI%20builds%20longer/near/215261215" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Noah Lev <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/consider.20storing.20commit.20CI.20builds.20longer.html#215261215">(Nov 01 2020 at 20:37)</a>:</h4>
<p>I guess they're billing themselves then <span aria-label="laughing" class="emoji emoji-1f606" role="img" title="laughing">:laughing:</span></p>



<a name="215261217"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/consider%20storing%20commit%20CI%20builds%20longer/near/215261217" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Pietro Albini <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/consider.20storing.20commit.20CI.20builds.20longer.html#215261217">(Nov 01 2020 at 20:37)</a>:</h4>
<p><a href="/user_uploads/4715/POtum1vqS6P9zctrgadfhCX2/2020-11-01-21-37-13.png">2020-11-01-21-37-13.png</a> <br>
oh, and it's not trending well btw ("thanks" to the faster builds)</p>
<div class="message_inline_image"><a href="/user_uploads/4715/POtum1vqS6P9zctrgadfhCX2/2020-11-01-21-37-13.png" title="2020-11-01-21-37-13.png"><img src="/user_uploads/4715/POtum1vqS6P9zctrgadfhCX2/2020-11-01-21-37-13.png"></a></div>



<a name="215261221"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/consider%20storing%20commit%20CI%20builds%20longer/near/215261221" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Pietro Albini <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/consider.20storing.20commit.20CI.20builds.20longer.html#215261221">(Nov 01 2020 at 20:37)</a>:</h4>
<p>that's just CI artifacts</p>



<a name="215261222"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/consider%20storing%20commit%20CI%20builds%20longer/near/215261222" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Joshua Nelson <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/consider.20storing.20commit.20CI.20builds.20longer.html#215261222">(Nov 01 2020 at 20:37)</a>:</h4>
<p>I can't imagine anyone trying to bisect things on raspberry pi</p>



<a name="215261223"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/consider%20storing%20commit%20CI%20builds%20longer/near/215261223" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Noah Lev <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/consider.20storing.20commit.20CI.20builds.20longer.html#215261223">(Nov 01 2020 at 20:37)</a>:</h4>
<p>Huh. What do you mean by "faster builds"</p>



<a name="215261267"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/consider%20storing%20commit%20CI%20builds%20longer/near/215261267" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Pietro Albini <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/consider.20storing.20commit.20CI.20builds.20longer.html#215261267">(Nov 01 2020 at 20:38)</a>:</h4>
<p><span class="user-mention" data-user-id="307537">@Camelid</span> now that we switched to GHA we're able to land more PRs in a single day</p>



<a name="215261268"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/consider%20storing%20commit%20CI%20builds%20longer/near/215261268" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Pietro Albini <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/consider.20storing.20commit.20CI.20builds.20longer.html#215261268">(Nov 01 2020 at 20:38)</a>:</h4>
<p>so we're storing more artifacts per day</p>



<a name="215261269"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/consider%20storing%20commit%20CI%20builds%20longer/near/215261269" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Noah Lev <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/consider.20storing.20commit.20CI.20builds.20longer.html#215261269">(Nov 01 2020 at 20:38)</a>:</h4>
<p>Yeah, we could consider just storing CI builds for a few major platforms (windows, macos, linux gnu)</p>



<a name="215261283"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/consider%20storing%20commit%20CI%20builds%20longer/near/215261283" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Joshua Nelson <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/consider.20storing.20commit.20CI.20builds.20longer.html#215261283">(Nov 01 2020 at 20:39)</a>:</h4>
<p><span class="user-mention silent" data-user-id="307537">Camelid</span> <a href="#narrow/stream/242791-t-infra/topic/consider.20storing.20commit.20CI.20builds.20longer/near/215261223">said</a>:</p>
<blockquote>
<p>Huh. What do you mean by "faster builds"</p>
</blockquote>
<p><a href="https://github.com/rust-lang/rust/pull/74620">https://github.com/rust-lang/rust/pull/74620</a></p>



<a name="215261296"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/consider%20storing%20commit%20CI%20builds%20longer/near/215261296" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Noah Lev <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/consider.20storing.20commit.20CI.20builds.20longer.html#215261296">(Nov 01 2020 at 20:39)</a>:</h4>
<p>Is there documentation on rust infra?</p>



<a name="215261297"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/consider%20storing%20commit%20CI%20builds%20longer/near/215261297" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Pietro Albini <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/consider.20storing.20commit.20CI.20builds.20longer.html#215261297">(Nov 01 2020 at 20:39)</a>:</h4>
<p>lol</p>



<a name="215261299"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/consider%20storing%20commit%20CI%20builds%20longer/near/215261299" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Noah Lev <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/consider.20storing.20commit.20CI.20builds.20longer.html#215261299">(Nov 01 2020 at 20:39)</a>:</h4>
<p>Because there are things like "auto" builds which I don't understand</p>



<a name="215261349"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/consider%20storing%20commit%20CI%20builds%20longer/near/215261349" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Noah Lev <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/consider.20storing.20commit.20CI.20builds.20longer.html#215261349">(Nov 01 2020 at 20:40)</a>:</h4>
<p>I just mean like a README somewhere</p>



<a name="215261350"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/consider%20storing%20commit%20CI%20builds%20longer/near/215261350" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Pietro Albini <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/consider.20storing.20commit.20CI.20builds.20longer.html#215261350">(Nov 01 2020 at 20:40)</a>:</h4>
<p>unfortunately docs were never written</p>



<a name="215261352"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/consider%20storing%20commit%20CI%20builds%20longer/near/215261352" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Joshua Nelson <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/consider.20storing.20commit.20CI.20builds.20longer.html#215261352">(Nov 01 2020 at 20:40)</a>:</h4>
<p><span class="user-mention" data-user-id="307537">@Camelid</span> there's <a href="https://forge.rust-lang.org/infra/index.html">https://forge.rust-lang.org/infra/index.html</a></p>



<a name="215261364"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/consider%20storing%20commit%20CI%20builds%20longer/near/215261364" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Pietro Albini <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/consider.20storing.20commit.20CI.20builds.20longer.html#215261364">(Nov 01 2020 at 20:40)</a>:</h4>
<p>I'm slowly writing them as I change stuff, but there is a ton which is not documented</p>



<a name="215261365"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/consider%20storing%20commit%20CI%20builds%20longer/near/215261365" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Noah Lev <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/consider.20storing.20commit.20CI.20builds.20longer.html#215261365">(Nov 01 2020 at 20:40)</a>:</h4>
<p>I've never noticed that!</p>



<a name="215261377"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/consider%20storing%20commit%20CI%20builds%20longer/near/215261377" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Pietro Albini <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/consider.20storing.20commit.20CI.20builds.20longer.html#215261377">(Nov 01 2020 at 20:41)</a>:</h4>
<p>note that the "service infra", "toolstate" and "team maintenance" are wildly out of date</p>



<a name="215261385"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/consider%20storing%20commit%20CI%20builds%20longer/near/215261385" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Pietro Albini <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/consider.20storing.20commit.20CI.20builds.20longer.html#215261385">(Nov 01 2020 at 20:41)</a>:</h4>
<p>the documentation for rust CI is at <a href="https://forge.rust-lang.org/infra/docs/rustc-ci.html">https://forge.rust-lang.org/infra/docs/rustc-ci.html</a></p>



<a name="215261388"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/consider%20storing%20commit%20CI%20builds%20longer/near/215261388" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Pietro Albini <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/consider.20storing.20commit.20CI.20builds.20longer.html#215261388">(Nov 01 2020 at 20:41)</a>:</h4>
<p>but I need to s/azure/gha/ on it</p>



<a name="215261394"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/consider%20storing%20commit%20CI%20builds%20longer/near/215261394" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Pietro Albini <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/consider.20storing.20commit.20CI.20builds.20longer.html#215261394">(Nov 01 2020 at 20:41)</a>:</h4>
<p>(well, and update a couple of things we actually changed during the migration)</p>



<a name="215261442"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/consider%20storing%20commit%20CI%20builds%20longer/near/215261442" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Pietro Albini <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/consider.20storing.20commit.20CI.20builds.20longer.html#215261442">(Nov 01 2020 at 20:42)</a>:</h4>
<p>so, back to the question at hand</p>



<a name="215261450"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/consider%20storing%20commit%20CI%20builds%20longer/near/215261450" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Pietro Albini <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/consider.20storing.20commit.20CI.20builds.20longer.html#215261450">(Nov 01 2020 at 20:42)</a>:</h4>
<p>there are a couple of ways we can reduce the amount of data we store</p>



<a name="215261464"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/consider%20storing%20commit%20CI%20builds%20longer/near/215261464" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Pietro Albini <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/consider.20storing.20commit.20CI.20builds.20longer.html#215261464">(Nov 01 2020 at 20:43)</a>:</h4>
<p>the first is a thing I'm investigating as part of the "change the whole release process" work I've been doing lately</p>



<a name="215261510"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/consider%20storing%20commit%20CI%20builds%20longer/near/215261510" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Noah Lev <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/consider.20storing.20commit.20CI.20builds.20longer.html#215261510">(Nov 01 2020 at 20:44)</a>:</h4>
<blockquote>
<p>"change the whole release process"</p>
</blockquote>
<p>?</p>



<a name="215261521"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/consider%20storing%20commit%20CI%20builds%20longer/near/215261521" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Pietro Albini <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/consider.20storing.20commit.20CI.20builds.20longer.html#215261521">(Nov 01 2020 at 20:44)</a>:</h4>
<p>I've been basically changing all of <code>promote-release</code></p>



<a name="215261534"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/consider%20storing%20commit%20CI%20builds%20longer/near/215261534" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Pietro Albini <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/consider.20storing.20commit.20CI.20builds.20longer.html#215261534">(Nov 01 2020 at 20:45)</a>:</h4>
<p>it's now on a separate repo, <a href="https://github.com/rust-lang/promote-release">https://github.com/rust-lang/promote-release</a>, it has integration tests, you can run it locally without production credentials, and it's now way faster</p>



<a name="215261540"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/consider%20storing%20commit%20CI%20builds%20longer/near/215261540" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Noah Lev <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/consider.20storing.20commit.20CI.20builds.20longer.html#215261540">(Nov 01 2020 at 20:45)</a>:</h4>
<p>Oh, I thought you meant no longer using the release train model or somethign</p>



<a name="215261548"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/consider%20storing%20commit%20CI%20builds%20longer/near/215261548" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Pietro Albini <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/consider.20storing.20commit.20CI.20builds.20longer.html#215261548">(Nov 01 2020 at 20:45)</a>:</h4>
<p>oh no it's just refactorings</p>



<a name="215261601"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/consider%20storing%20commit%20CI%20builds%20longer/near/215261601" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Pietro Albini <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/consider.20storing.20commit.20CI.20builds.20longer.html#215261601">(Nov 01 2020 at 20:46)</a>:</h4>
<p>and now that we can just throw more cores at the release process we can actually consider storing just <code>.xz</code>s on the CI bucket and recompress them into <code>.gz</code> during the release</p>



<a name="215261603"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/consider%20storing%20commit%20CI%20builds%20longer/near/215261603" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Pietro Albini <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/consider.20storing.20commit.20CI.20builds.20longer.html#215261603">(Nov 01 2020 at 20:46)</a>:</h4>
<p>which should cut our storage bill by more than 50%</p>



<a name="215261630"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/consider%20storing%20commit%20CI%20builds%20longer/near/215261630" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Noah Lev <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/consider.20storing.20commit.20CI.20builds.20longer.html#215261630">(Nov 01 2020 at 20:46)</a>:</h4>
<p>I've looked at some of the changes and I just get scared when it's all run by a shell script <span aria-label="sweat smile" class="emoji emoji-1f605" role="img" title="sweat smile">:sweat_smile:</span> </p>
<p>I'm impressed that you're able to work on it and refactor it successfully</p>



<a name="215261641"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/consider%20storing%20commit%20CI%20builds%20longer/near/215261641" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Pietro Albini <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/consider.20storing.20commit.20CI.20builds.20longer.html#215261641">(Nov 01 2020 at 20:47)</a>:</h4>
<p><span class="user-mention" data-user-id="307537">@Camelid</span> well the release process is not shell scripts <span aria-label="sweat smile" class="emoji emoji-1f605" role="img" title="sweat smile">:sweat_smile:</span></p>



<a name="215261642"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/consider%20storing%20commit%20CI%20builds%20longer/near/215261642" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Noah Lev <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/consider.20storing.20commit.20CI.20builds.20longer.html#215261642">(Nov 01 2020 at 20:47)</a>:</h4>
<p><span class="user-mention silent" data-user-id="121055">Pietro Albini</span> <a href="#narrow/stream/242791-t-infra/topic/consider.20storing.20commit.20CI.20builds.20longer/near/215261601">said</a>:</p>
<blockquote>
<p>and now that we can just throw more cores at the release process we can actually consider storing just <code>.xz</code>s on the CI bucket and recompress them into <code>.gz</code> during the release</p>
</blockquote>
<p>Why does that decrease storage size?</p>



<a name="215261646"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/consider%20storing%20commit%20CI%20builds%20longer/near/215261646" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Pietro Albini <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/consider.20storing.20commit.20CI.20builds.20longer.html#215261646">(Nov 01 2020 at 20:47)</a>:</h4>
<p>and the first thing I did was to add CI for that and regression tests</p>



<a name="215261690"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/consider%20storing%20commit%20CI%20builds%20longer/near/215261690" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Pietro Albini <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/consider.20storing.20commit.20CI.20builds.20longer.html#215261690">(Nov 01 2020 at 20:48)</a>:</h4>
<p><span class="user-mention silent" data-user-id="307537">Camelid</span> <a href="#narrow/stream/242791-t-infra/topic/consider.20storing.20commit.20CI.20builds.20longer/near/215261642">said</a>:</p>
<blockquote>
<p><span class="user-mention silent" data-user-id="121055">Pietro Albini</span> <a href="#narrow/stream/242791-t-infra/topic/consider.20storing.20commit.20CI.20builds.20longer/near/215261601">said</a>:</p>
<blockquote>
<p>and now that we can just throw more cores at the release process we can actually consider storing just <code>.xz</code>s on the CI bucket and recompress them into <code>.gz</code> during the release</p>
</blockquote>
<p>Why does that decrease storage size?</p>
</blockquote>
<p>at the moment we're storing the same tarball twice on the CI bucket, both with <code>.gz</code> and <code>.xz</code> compression</p>



<a name="215261692"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/consider%20storing%20commit%20CI%20builds%20longer/near/215261692" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Noah Lev <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/consider.20storing.20commit.20CI.20builds.20longer.html#215261692">(Nov 01 2020 at 20:48)</a>:</h4>
<p><span class="user-mention silent" data-user-id="121055">Pietro Albini</span> <a href="#narrow/stream/242791-t-infra/topic/consider.20storing.20commit.20CI.20builds.20longer/near/215261641">said</a>:</p>
<blockquote>
<p><span class="user-mention silent" data-user-id="307537">Camelid</span> well the release process is not shell scripts <span aria-label="sweat smile" class="emoji emoji-1f605" role="img" title="sweat smile">:sweat_smile:</span></p>
</blockquote>
<p>Sorry, I meant the CI scripts: <a href="https://github.com/rust-lang/rust/blob/master/src/ci/github-actions/ci.yml">https://github.com/rust-lang/rust/blob/master/src/ci/github-actions/ci.yml</a></p>



<a name="215261697"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/consider%20storing%20commit%20CI%20builds%20longer/near/215261697" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Noah Lev <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/consider.20storing.20commit.20CI.20builds.20longer.html#215261697">(Nov 01 2020 at 20:48)</a>:</h4>
<p>Couldn't we just store <code>.gz</code>s and then not need to recompress them?</p>



<a name="215261699"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/consider%20storing%20commit%20CI%20builds%20longer/near/215261699" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Noah Lev <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/consider.20storing.20commit.20CI.20builds.20longer.html#215261699">(Nov 01 2020 at 20:48)</a>:</h4>
<p>I think I'm missing something <span aria-label="smile" class="emoji emoji-1f642" role="img" title="smile">:smile:</span></p>



<a name="215261762"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/consider%20storing%20commit%20CI%20builds%20longer/near/215261762" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Pietro Albini <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/consider.20storing.20commit.20CI.20builds.20longer.html#215261762">(Nov 01 2020 at 20:50)</a>:</h4>
<p>the releases we ship (nightly, beta, stable) need to have the tarballs compressed both in <code>.gz</code> (since it's readable virtually anywhere) and <code>.xz</code> (since it's better at compressing stuff)</p>



<a name="215261781"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/consider%20storing%20commit%20CI%20builds%20longer/near/215261781" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Pietro Albini <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/consider.20storing.20commit.20CI.20builds.20longer.html#215261781">(Nov 01 2020 at 20:51)</a>:</h4>
<p>so what we do right now is, CI creates both the <code>.gz</code> and the <code>.xz</code> versions of the same tarball, and uploads both to the artifacts bucket (the one that's 71TB in size)</p>



<a name="215261792"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/consider%20storing%20commit%20CI%20builds%20longer/near/215261792" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Pietro Albini <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/consider.20storing.20commit.20CI.20builds.20longer.html#215261792">(Nov 01 2020 at 20:51)</a>:</h4>
<p>then the release process grabs the ones for the latest commit in the relevant branch, signs them and releases them</p>



<a name="215261835"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/consider%20storing%20commit%20CI%20builds%20longer/near/215261835" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Pietro Albini <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/consider.20storing.20commit.20CI.20builds.20longer.html#215261835">(Nov 01 2020 at 20:52)</a>:</h4>
<p>but the artifact bucket contains <em>all</em> builds for the 167 days we're storing artifacts for, not just the ones we released</p>



<a name="215261850"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/consider%20storing%20commit%20CI%20builds%20longer/near/215261850" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Noah Lev <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/consider.20storing.20commit.20CI.20builds.20longer.html#215261850">(Nov 01 2020 at 20:52)</a>:</h4>
<p>Ah, I think I understand. Basically you're saying <code>xz</code> is better, but it's not as widely accepted, so we release <code>xz</code> and <code>gz</code>?</p>



<a name="215261860"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/consider%20storing%20commit%20CI%20builds%20longer/near/215261860" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Pietro Albini <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/consider.20storing.20commit.20CI.20builds.20longer.html#215261860">(Nov 01 2020 at 20:53)</a>:</h4>
<p>so we can save more than 50% of the storage if we only upload the <code>.xz</code> to the artifacts bucket (since it's smaller), and then recompress the tarballs during the release process just for the artifacts we're actually releasing</p>



<a name="215261861"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/consider%20storing%20commit%20CI%20builds%20longer/near/215261861" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Pietro Albini <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/consider.20storing.20commit.20CI.20builds.20longer.html#215261861">(Nov 01 2020 at 20:53)</a>:</h4>
<p>yes</p>



<a name="215262016"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/consider%20storing%20commit%20CI%20builds%20longer/near/215262016" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Noah Lev <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/consider.20storing.20commit.20CI.20builds.20longer.html#215262016">(Nov 01 2020 at 20:58)</a>:</h4>
<p>Okay, thanks for explaining! :)</p>



<a name="215345426"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/consider%20storing%20commit%20CI%20builds%20longer/near/215345426" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Josh Triplett <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/consider.20storing.20commit.20CI.20builds.20longer.html#215345426">(Nov 02 2020 at 16:48)</a>:</h4>
<p>I honestly think we could get away with not providing .gz for releases either. But before we do that I would want to switch from xz to zstd (better compression and faster decompression).</p>



<a name="215345850"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/consider%20storing%20commit%20CI%20builds%20longer/near/215345850" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Josh Triplett <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/consider.20storing.20commit.20CI.20builds.20longer.html#215345850">(Nov 02 2020 at 16:51)</a>:</h4>
<p>Cutting storage in half sounds great.</p>



<a name="215347726"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/consider%20storing%20commit%20CI%20builds%20longer/near/215347726" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Pietro Albini <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/consider.20storing.20commit.20CI.20builds.20longer.html#215347726">(Nov 02 2020 at 17:04)</a>:</h4>
<p><span class="user-mention" data-user-id="239881">@Josh Triplett</span> well, if we remove <code>.gz</code> we break rustup</p>



<a name="215347796"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/consider%20storing%20commit%20CI%20builds%20longer/near/215347796" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Josh Triplett <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/consider.20storing.20commit.20CI.20builds.20longer.html#215347796">(Nov 02 2020 at 17:05)</a>:</h4>
<p>rustup can use both, I thought.</p>



<a name="215347811"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/consider%20storing%20commit%20CI%20builds%20longer/near/215347811" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Pietro Albini <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/consider.20storing.20commit.20CI.20builds.20longer.html#215347811">(Nov 02 2020 at 17:05)</a>:</h4>
<p>since the manifest format requires the <code>.gz</code> URL to be available if the component is available</p>



<a name="215347856"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/consider%20storing%20commit%20CI%20builds%20longer/near/215347856" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Pietro Albini <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/consider.20storing.20commit.20CI.20builds.20longer.html#215347856">(Nov 02 2020 at 17:05)</a>:</h4>
<p>if we only provide <code>.xz</code> (or <code>.zstd</code>) rustup will think all the components are unavailable</p>



<a name="215347858"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/consider%20storing%20commit%20CI%20builds%20longer/near/215347858" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Pietro Albini <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/consider.20storing.20commit.20CI.20builds.20longer.html#215347858">(Nov 02 2020 at 17:05)</a>:</h4>
<p>iirc</p>



<a name="215347949"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/consider%20storing%20commit%20CI%20builds%20longer/near/215347949" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Josh Triplett <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/consider.20storing.20commit.20CI.20builds.20longer.html#215347949">(Nov 02 2020 at 17:06)</a>:</h4>
<p>But in any case, rustup could learn to handle that, which we would do before making any such change. (And we would also have to ensure rustup starts updating itself <em>before</em> updating rust.)</p>



<a name="215348019"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/consider%20storing%20commit%20CI%20builds%20longer/near/215348019" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Josh Triplett <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/consider.20storing.20commit.20CI.20builds.20longer.html#215348019">(Nov 02 2020 at 17:06)</a>:</h4>
<p>(zstd uses <code>.zst</code>)</p>



<a name="215348044"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/consider%20storing%20commit%20CI%20builds%20longer/near/215348044" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Mara <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/consider.20storing.20commit.20CI.20builds.20longer.html#215348044">(Nov 02 2020 at 17:07)</a>:</h4>
<p>what happens if the .gz is just a redirect to the .xz or .zst? would it handle that correctly?</p>



<a name="215349283"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/consider%20storing%20commit%20CI%20builds%20longer/near/215349283" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Pietro Albini <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/consider.20storing.20commit.20CI.20builds.20longer.html#215349283">(Nov 02 2020 at 17:15)</a>:</h4>
<blockquote>
<p>(And we would also have to ensure rustup starts updating itself before updating rust.)</p>
</blockquote>
<p>cc <span class="user-mention" data-user-id="223910">@Daniel Silverstone</span>, my understanding is that doesn't work on windows</p>



<a name="215349648"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/consider%20storing%20commit%20CI%20builds%20longer/near/215349648" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Daniel Silverstone <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/consider.20storing.20commit.20CI.20builds.20longer.html#215349648">(Nov 02 2020 at 17:17)</a>:</h4>
<p>I <em>think</em> we support xz on Windows (and we certainly would support zstd on Windows)</p>



<a name="215349769"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/consider%20storing%20commit%20CI%20builds%20longer/near/215349769" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Daniel Silverstone <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/consider.20storing.20commit.20CI.20builds.20longer.html#215349769">(Nov 02 2020 at 17:18)</a>:</h4>
<p>As for requiring .gz - if that's currently the case then we need to adjust that too, probably as part of the add zstd work.</p>



<a name="215349863"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/consider%20storing%20commit%20CI%20builds%20longer/near/215349863" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Pietro Albini <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/consider.20storing.20commit.20CI.20builds.20longer.html#215349863">(Nov 02 2020 at 17:19)</a>:</h4>
<p><span class="user-mention" data-user-id="223910">@Daniel Silverstone</span> oh, I mean updating rustup before it downloads other stuff</p>



<a name="215349957"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/consider%20storing%20commit%20CI%20builds%20longer/near/215349957" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Pietro Albini <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/consider.20storing.20commit.20CI.20builds.20longer.html#215349957">(Nov 02 2020 at 17:20)</a>:</h4>
<p><span class="user-mention" data-user-id="223910">@Daniel Silverstone</span> <a href="#narrow/stream/242791-t-infra/topic/consider.20storing.20commit.20CI.20builds.20longer/near/215349769">said</a>:</p>
<blockquote>
<p>As for requiring .gz - if that's currently the case then we need to adjust that too, probably as part of the add zstd work.</p>
</blockquote>
<p><a href="https://github.com/rust-lang/rustup/blob/c0b6dc510c45f0ae1f9a4f67b35ad62cf34bee21/src/dist/manifest.rs#L54-L60">https://github.com/rust-lang/rustup/blob/c0b6dc510c45f0ae1f9a4f67b35ad62cf34bee21/src/dist/manifest.rs#L54-L60</a></p>



<a name="215350973"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/consider%20storing%20commit%20CI%20builds%20longer/near/215350973" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Daniel Silverstone <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/consider.20storing.20commit.20CI.20builds.20longer.html#215350973">(Nov 02 2020 at 17:27)</a>:</h4>
<p>Can't pre-update rustup for Windows binary execution reasons currently (though I may have a way around that particular horrorpile)<br>
And yeah that struct is heinous <span aria-label="grinning" class="emoji emoji-1f600" role="img" title="grinning">:grinning:</span></p>



<hr><p>Last updated: Aug 07 2021 at 22:04 UTC</p>
</html>